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complete protocols ask the subjects to verbalize their
intentions, what their goals are, and what current plans
they have about reaching their goals. Other behavior is
directly observable; thoughts and plans typically are
not. This method has been used by Mack et al. (1983),
Carroll and Mack (1982), and Card et al. (1980a) in their
studies of skilled text editing. More complete descrip-
tions of the technique and its advantages and disadvan-
tages can be found in Lewis (1982) , Olson et al. (1984),
and Ericsson and Simon (1980).

A third kind of data collected in evaluation sessions
is the users' opinions about the system's ease of use
and functionality. A common instrument used to scale
users' global attitudes about the system is the evalua-
tion component of Osgood et al.'s (1957) Semantic
Differential (see Good, 1982, for an example of its
use). Questionnaires and interviews also tap users'
reactions to particular components of the'system. One
problem with users' reports, however, is that they are
typically distorted by their experience with other,
similar systems. Or a user may have difficulty separating
components of the system such; for example, a user who
has a very difficult time using a system may report that
he or she likes it a great deal, recognizing how much
easier it is to perform the task on a computer compared
with previous manual methods.

Redesign

Typically as the prototype of the original design is
tested, errors are found and revisions suggested. The
methods appropriate to the initial design are appropriate
also at the stage of redesign. This part of the design
process iterates through "fixing" and "testing" until
either an acceptable level of performance is reached or
the deadline for developing the system is reached.

IMPLEMENTATIONS  MONITORING CONTINUED PERFORMANCE

Just as data were collected in the original conception
and analysis phase of product development, data are col-
lected on the system as implemented. At this stage,
activity analyses, diaries, logging and metering, and
questionnaires and interviews are all appropriate methods
for assessing whether the product as designed is perform-